Real world approaches for multilingual and non-native speech recognition

This thesis proposes a scalable architecture for multilingual speech recognition on embedded devices. In theory multiple languages can be recognized just as one language. However, current state of the art speech recognition systems are based on statistical models with many parameters. Extending such models to multiple languages requires more resources. Therefore a lot of research in the area of multilingual speech recognition has proposed techniques to reduce this need for more resources through parameter tying across languages. After an evaluation of the previous work, this thesis was able to show that tying at the density level offers the greatest flexibility for the design of a multilingual acoustic model. Furthermore, there were also hints in the literature that densities from the native language of the speakers can be useful for the modeling of non-native accents of speakers. Based on these findings, this thesis developed an algorithm for the creation of Multilingual Weighted Codebooks (MWCs) that adds Gaussians from the spoken languages to the native language codebook (= set of Gaussians) of the speaker. A key advantage of this algorithm is that it optimally models the native language of the speaker, which is not the case for most of the previous work. The results prove the effectiveness of the MWC algorithm, both for native and non-native speech, but the disadvantage of this algorithm is that it increases the training effort exponentially with the number of languages considered. The answer to this problem was found in projections between Gaussian spaces. These projections allow to generate multilingual models within fractions of a second from monolingual speech recognizers. Due to this, the problem of training effort was eliminated, as there is no longer the need to provide all possible acoustic models. Instead, it is possible to determine the languages that are needed on the embedded system and to generate the required acoustic model online. Of course, this large reduction in time causes a reduction in performance, but a combination of the MWC algorithm and the on-the-fly creation of new models leads to a scalable architecture that can recognize all languages with good performance. At the same time, the target resources are almost independent of the number of languages. Finally, this thesis also compared several additional approaches for the optimal recognition of non-native accented speech. As the literature indicated, the use of the native language codebook of the speaker in the MWC algorithm already gave a significant improvement over monolingual systems. From the other tested algorithms, only the adaptation with additional non-native development data could outperform the baseline of native language codebooks.